Methods of deblurring image and recording mediums having the same recorded thereon

ABSTRACT

A method of deblurring an image by which blur can be easily and rapidly eliminated from one image and the quality of the image can be improved is provided. The method includes receiving a blurred image, an image estimation step of estimating a non-blurred image from the blurred image, a blur information estimation step of estimating blur information from the blurred image and the estimated non-blurred image, and a deblurring step of deblurring the blurred image based on the blurred image and the estimated blur information, wherein the image estimation step and the blur information estimation step are iteratively performed. Thus, blur can be rapidly and effectively eliminated from one image, thereby improving the quality of an image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit under 35 U.S.C. §119(a) of KoreanPatent Application No. 2008-138195, filed on Dec. 31, 2008, thedisclosure of which is incorporated herein in its entirety by reference.

BACKGROUND

1. Field

The present invention relates to image processing methods, and moreparticularly, to methods of deblurring an image, and recording mediumshaving the same recorded thereon.

2. Description of the Related Art

A blur phenomenon may often occur in a process of acquiring an imageusing an image acquisition device and is a cause of degraded quality ofan image.

In order to acquire an image using a device, such as a camera, in anenvironment including an insufficient amount of light, such as, inside adark room or outside a room in the evening, a sufficient amount of lightis necessary to obtain a strong image. For the sufficient amount oflight, an image sensor must be exposed to light for a long time. Thelong exposure, however, causes an acquired image to be blurred due to ashaken image sensor.

Although elimination of a blur phenomenon has been studied a great deal,it is still difficult to eliminate blur from an image. This is becauseestimating and eliminating blur from an image require more newinformation than given information.

In order to resolve the problem, conventional solutions use severalimages, require additional information such as an exposure time of asensor, or assume a limited shape of blur, such as a blur shape limitedto a linear motion that can be represented by a two-dimensional vector.

Ytizhakey et al. (YTIZHZKEY, Y., MOR, I., A., and KOPEIKA, N. S., 1998,Direct Method For Restoration of Motion-blurred Images. Journal of Opt.Soc. Am. A. 15, 6, 1512-1519) estimates blur that can be representedusing a 2D vector on the assumption that an image has isotropy. Rav-Achaand Peleg (RAV-ACHA, A. and PELEG, S., 2005, Two Motion-blurred ImagesAre Better Than One. Pattern Recognition Letters 26, 311-317.) proposeda method of estimating blur using two blurred images. Yuan et al. (YUAN,L., SUN, J., QUAN, L., and SHUM, H.-Y., 2007, Image Deblurring WithBlurred/Noisy Image Pairs. ACM Trans. Graphics 26, 3, 1.) proposed amethod of estimating and eliminating blur using a noisy non-blurredimage and a blurred image.

Money and Kang (MONEY, J. H. and KANG, S. H., 2008, Total VariationMinimizing Blind Deconvolution with Shock Filter Reference. Image andVision Computing 26, 2, 302-314.) proposed a method of estimatingGaussian blur and blur capable of being represented using a 2D vector,by applying shock filtering to a blurred image to restore sharp edgesand then using the sharp edges.

In recent years, methods of estimating general blur rather than motionblur capable of being represented using a small number of parametersfrom one image and eliminating the general blur have been introduced.Fergus et al. (FERGUS, R., SINGH, B., HERTZMANN, A., ROWEIS, S. T., andFREEMAN, W., 2006, Removing Camera Shake From A Single Photograph. ACMTrans. Graphics 25, 787-794.) proposed a method of estimating blur usinga statistical characteristic of a general image. Jia (JIA, J., 2007,Single Image Motion Deblurring Using Transparency. In Proc. CVPR 2007,1-8.) proposed a method of finding information on a blur occurrenceregion in an image using an alpha matte scheme and then deblurring theimage. However, in the method proposed by Fergus et al. an excellentresult is difficult to derive and the method consumes much time due to acomplex statistical model, and for the method proposed by Jia excellentmatte must be obtained for a satisfactory result due to its highdependence on a result of the alpha matte scheme.

Shan et al. (SHAN, Q., JIA, J., and AGARWALA, A., 2008, High-QualityMotion Deblurring From A Single Image. ACM Trans. Graphics 27, 73.)proposed a method of estimating and eliminating blur by suggesting thestatistical characteristic of the general image proposed by Fergus etal. in a form enabling easy calculation and using the statisticalcharacteristic. However, the method is impractical because a processingtime from a few minutes to about ten minutes or more is required toprocess one image.

SUMMARY

The present invention is directed to a method by which one image can beeasily and rapidly deblurred and the quality of an image can beimproved.

The present invention is also directed to a recording medium having amethod recorded thereon by which one image can be easily and rapidlydeblurred and the quality of an image can be improved.

In example embodiments, a method of deblurring an image includes:receiving a blurred image; an image estimation step of estimating anon-blurred image from the blurred image; a blur information estimationstep of estimating blur information from the blurred image and theestimated non-blurred image; and a deblurring step of deblurring theblurred image based on the blurred image and the estimated blurinformation. Here, the image estimation step and the blur informationestimation step are iteratively performed. The image estimation step mayinclude: a gradient information prediction step of predicting gradientinformation for the blurred image; and a first deconvolution step ofperforming first deconvolution based on the estimated blur informationand the blurred image. The gradient information prediction step mayinclude: applying bidirectional filtering to the blurred image; applyingshock filtering to the bidirectional filtered image; calculating agradient map for the shock filtered image; and applying a thresholdvalue to the calculated gradient map. Applying a threshold value to thecalculated gradient map may include: creating a histogram based on adirection and a size of the calculated gradient; setting a size of agradient capable of including as many pixels as a predetermined multipleor more of a maximum value of vertical and horizontal sizes of the blurinformation corresponding to each direction included in the createdhistogram, as a threshold value; and applying the set threshold value tothe gradient map to truncate the gradient. Applying the set thresholdvalue to the gradient map to truncate the gradient may include setting agradient smaller than the threshold value to about zero. The blurinformation estimation step may include estimating the blur informationbased on the blurred image and the truncated gradient. The blurinformation estimation step may include estimating the blur informationusing an energy function including only image derivatives without apixel value. The image estimation step and the blur informationestimation step may be iteratively performed while changing resolutionsof the blurred image and the estimated non-blurred image.

In example embodiments, a recording medium having a program ofinstructions recorded thereon is provided. The program performs:receiving a blurred image; an image estimation step of estimating anon-blurred image from the blurred image; a blur information estimationstep of estimating blur information from the blurred image and theestimated non-blurred image; and a deblurring step of deblurring theblurred image based on the blurred image and the estimated blurinformation. Here, the image estimation step and the blur informationestimation step are iteratively performed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate embodiments of the invention, andtogether with the description serve to explain aspects of the invention.

FIG. 1 is a flowchart of a method of deblurring an image according to anexemplary embodiment of the present invention;

FIG. 2 illustrates a deblurred image according to a number of iterationsin a method of deblurring an image according to an exemplary embodimentof the present invention in FIG. 1;

FIG. 3 is a flowchart of a prediction step in FIG. 1;

FIG. 4 illustrates kernels estimated in different scales in the methodof deblurring an image according to an exemplary embodiment of thepresent invention;

FIG. 5 illustrates a comparison in a convergence speed of blur kernelestimation between a method of estimating a blur kernel according to anexemplary embodiment of the present invention and a conventional methodof estimating a blur kernel;

FIG. 6 illustrates the accuracy of blur kernel estimation according toan exemplary embodiment of the present invention;

FIG. 7 illustrates images deblurred using a method of estimating blur ofan image according to an exemplary embodiment of the present invention;

FIG. 8 illustrates actual images deblurred using a method of estimatingblur of an image according to an exemplary embodiment of the presentinvention; and

FIG. 9 illustrates a processing time for deblurring images shown in FIG.8.

DETAILED DESCRIPTION OF EMBODIMENTS

Accordingly, while example embodiments are capable of variousmodifications and alternative forms, embodiments thereof are shown byway of example in the drawings and will herein be described in detail.It should be understood, however, that there is no intent to limitexample embodiments to the particular forms disclosed, but on thecontrary, example embodiments are to cover all modifications,equivalents, and alternatives falling within the scope of the invention.Like numbers refer to like elements throughout the description of thefigures.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another. For example, a first element could be termed asecond element, and, similarly, a second element could be termed a firstelement, without departing from the scope of example embodiments. Asused herein, the term “and/or” includes any and all combinations of oneor more of the associated listed items.

It will be understood that when an element is referred to as being“connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments. As used herein, the singular forms “a,” “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises,” “comprising,” “includes” and/or “including,” when usedherein, specify the presence of stated features, integers, steps,operations, elements and/or components, but do not preclude the presenceor addition of one or more other features, integers, steps, operations,elements, components and/or groups thereof.

Unless otherwise defined, all terms used therein including technical orscientific terms have meanings understood by those skilled in the art.Terms generally defined in dictionaries should be construed as having ameaning on a context of related schemes, and not as having an abnormallyor inordinately formal meaning, unless clearly defined.

Hereinafter, preferred exemplary embodiments of the present inventionwill be described in greater detail with reference to the accompanyingdrawings.

In general, motion blur may be represented using Equation 1:

B=K*L+N  Equation 1

where B denotes a blurred image, K denotes a motion blur kernel or apoint spread function (PSF) indicating blur information for an image,and L denotes a latent image, i.e., a strong non-blurred image. Also, Ndenotes an unknown noise introduced in an image acquisition process,and * denotes a convolution operator.

For blind deconvolution of Equation 1, L and K must be optimized in aniteration process. Also, the latent image and the blur kernel can beestimated using Equations 2 and 3:

L′=argmin_(L) {∥B−K*L∥+ρ _(L)(L)}  Equation 2

K′=argmin_(K) {∥B−K*L∥+ρ _(K)(L)}  Equation 3

where ∥B−K*L∥ denotes a data fitting term, and ρ_(L) and ρ_(K) denoteregularization terms.

Iterative optimization is intended to gradually refine the accuracy ofthe blur kernel K. A final deblurred image is obtained by performing adeconvolution operation on the final blur kernel K and the given blurredimage B. A latent image estimated during the iterative optimization doesnot have a direct influence on the final deblurred image but onrefinement of the motion blur kernel K, such that the latent image hasan indirect influence on the final deblurred image.

The iterative optimization process for latent image estimation enablessharp edges to be restored from the latent image and a noise reductioncharacteristic to be obtained in a smooth region, which is used toestimate a more accurate motion blur kernel. Even when it is assumedthat a given image includes constant blur, the more accurate motion blurkernel can be acquired around sharp edges. For example, the motion blurkernel cannot be estimated in a region having constant intensity, butsince a natural image generally includes sharp edges, the motion blurkernel may be effectively predicted through the edges restored in thelatent image estimation process.

In a general natural image, a smooth region occupies a larger space thana sharp edge region, such that suppression of noises in the smoothregion is critical. Noises not suppressed in the smooth region maygreatly affect the data fitting term of Equation 3, which degrades theaccuracy of estimation of the blur kernel from the sharp edges.

In a process of solving Equation 2, conventional methods for restoringsharp edges and suppressing noises use non-linear optimization withcomplex calculations. Also, a conventional method of estimating a blurkernel using Equation 3 requires operation of a great number of matrixesand vectors. As a result, the conventional method for iterativelyperforming blind deconvolution for estimation of a latent image and ablur kernel has a shortcoming of a great amount of calculation.

To overcome the shortcomings of the conventional methods as describedabove, the method of deblurring an image according to an exemplaryembodiment of the present invention provides fast blind deconvolution byreducing a calculation amount in a process of estimating a latent imageand a blur kernel.

In the method of deblurring an image according to an exemplaryembodiment of the present invention, it is assumed that a latent imageincludes sufficient sharp edges to improve an estimation speed of thelatent image, and sharp edges are restored using an image filter and anoise suppression characteristic is used instead of optimizing anon-linear regularization term having high calculation complexity inEquation 2.

Also, a blur kernel estimation process results in an improved speed of anumerical optimization process of Equation 3 by excluding a pixel value.

In the method of deblurring an image according to an exemplaryembodiment of the present invention, the latent image estimation processincludes two steps: simple deconvolution and prediction.

In the simple deconvolution step, given a blurred image B and a blurkernel K, blur is first eliminated using simple and fast deconvolutionhaving Gaussian prior in order to estimate the latent image L. Thelatent image L includes noises in smooth edges and a smooth region dueto characteristics of the Gaussian prior.

In a step of predicting the latent image L, sharp edges are restored andnoises are eliminated using an image filter to obtain an estimatedlatent image L′ with refined accuracy. Here, the sharp edges and thesuppressed noises in the smooth region of the latent image are used ascritical characteristics when the latent image is used to estimate theblur kernel, such that the estimated latent image L′ can provide ahigh-quality latent image for accurate estimation of the blur kerneleven if the simple deconvolution degrades the quality of the image.

The blur kernel estimation step uses a conjugate gradient (CG) to solveEquation 3. Also, gradient calculation for an energy function isperformed several times in solving Equation 3.

Since the gradient calculation involves a large matrix and vectormultiplication, it is very complex. A calculation speed ofmultiplication associated with the convolution operation can be improvedby using a fast Fourier transform.

However, it is necessary to properly adjust an image boundary thatobstructs direct concatenation of the fast Fourier transform insequentially performing the fast Fourier transform.

A number of image boundary adjustments and fast Fourier transforms canbe greatly reduced by formulating Equation 3 to obtain an energyfunction having only image derivatives. Also, the energy functionimproves a processing speed of the conjugate gradient (CG), provides awell condition having a small condition number to a calculation systemderived from Equation 3, and improves a convergence speed.

FIG. 1 is a flowchart of a method of deblurring an image according to anexemplary embodiment of the present invention.

Referring to FIG. 1, a blurred image is first provided (step 110).

A prediction step 120 is located at an initial stage of an iterativeloop to provide an initial value of a latent image L for estimation of ablur kernel when the blurred image is provided.

In the prediction step, a gradient map {Px, Py} is calculated in x and ydirections of the latent image L to predict prominent edges from thelatent image in which noises are suppressed in the smooth region.

In iterations of steps 120 to 150, the estimated latent image L acquiredby the deconvolution step in the previous iteration is input to theprediction step although it is not an initial input.

The blur kernel estimation step 130 estimates the blur kernel K usingthe gradient map {Px, Py} predicted through the calculation in step 120and the blurred image B.

In a deconvolution step 140, the blur kernel K and the blurred image Bare used to estimate the latent image L. Here, the estimated latentimage L is input to the prediction step 120 in a next iteration.

In the method of deblurring an image according to an exemplaryembodiment of the present invention, steps 120 and 130 are iterativelyperformed to estimate the blur kernel K and the latent image L moreeffectively, thereby improving the accuracy of the estimation.

In the method of deblurring an image according to an exemplaryembodiment of the present invention, grayscale versions of the blurredimage B and the latent image L are used in the iteration process forupdating the accuracy of the blur kernel K and the latent image L.

After the final blur kernel K is obtained through the iterations, finaldeconvolution is performed on the final blur kernel K and each colorchannel of the blurred image B to obtain a final deblurred image (step150).

FIG. 2 illustrates a deblurred image according to a number of theiterations in the method of deblurring an image according to anexemplary embodiment of the present invention shown in FIG. 1.

FIG. 2( a) illustrates an original blurred image, and FIG. 2( b)illustrates a final deblurred image according to an exemplary embodimentof the present invention.

FIGS. 2( c) to 2(e) illustrate results of iteratively performing steps120 to 140 in FIG. 1 once, three times, and five times, respectively,i.e., deconvolution results including a predicted gradient map {Px, Py}and an estimated blur kernel. In FIG. 2, Poisson reconstruction was usedfor visualization in the predicted gradient map {Px, Py}.

As shown in FIG. 2, in the method of deblurring an image according to anexemplary embodiment of the present invention, as a number of theiterations of steps 120 to 140 in FIG. 1 increases (the 5th iteration inFIG. 2( e)), the latent image can be estimated more accurately and amore accurate blur kernel can be predicted based on the estimated latentimage. The more accurately estimated blur kernel may be used to obtain afinal deblurred image (b).

A method of deblurring an image according to an exemplary embodiment ofthe present invention will now be described in greater detail.

First, in the prediction step 120 in FIG. 1, the image gradient map {Px,Py} is estimated by leaving only prominent edges in the latent image Land setting a gradient for other regions to 0. Convolution of gradient 0is always zero irrespective of the estimated blur kernel, such that onlyprominent edges affect the blur kernel optimization in the blur kernelestimation step 130.

The prediction step uses a shock filter to restore strong edges. Theshock filter is an efficient tool for strengthening features of an imageby restoring sharp edges from the blurred image.

The shock filter may be represented using Equation 4:

I _(t+1) =I _(t)−sign(ΔI _(t))∥∇I _(t) ∥dt  Equation 4

where I_(t) denotes an image at a time t, and ΔI_(t) and ∇I_(t) denotethe Laplacian and gradient of I_(t), respectively. Also, dt denotes atime step for single evolution.

FIG. 3 is a flowchart of the prediction step in FIG. 1.

Referring to FIG. 3, the prediction step may include applyingbidirectional filtering, applying shock filtering, and applying agradient threshold value.

First, bidirectional filtering is applied to suppress noises and smalldetails that may be present in the estimated current latent image L(step 121). Here, an image size capable of being supported inbidirectional filtering may be fixed to, e.g., 5×5.

Shock filtering is then applied to restore sharp edges in the estimatedlatent image L (step 123). In a latent image L′ that is a shock-filteredresultant image, contrast of edges is improved but noises are increased.

In order to eliminate the noises, a gradient map {∂_(x)L′, ∂_(y)L′} ofthe latent image L′ is calculated (step 125) and a threshold value isapplied to the calculated gradient map to truncate the gradient (step127). The truncated gradient map {P_(x), P_(y)} becomes a final outputof the prediction step. Here, the gradient may be truncated, forexample, by setting a gradient smaller than the threshold value to 0.

The threshold value for truncating the gradient is obtained by dividinga direction of the gradient into intervals of a predetermined-angle(e.g., 45°), creating a histogram according to a gradient size for eachdirection (where, angles between 180° and 360° are regarded as those indirections in which only signs are different and sizes equals), andsetting a size of a gradient that can include as many pixels as apredetermined multiple (e.g., double) or more of a maximum value ofvertical and horizontal sizes of the blur kernel for each direction, asthe threshold value.

In a deconvolution step 140, the latent image L is estimated based on agiven blur kernel K and a provided blurred image B. Here, the latentimage L is estimated using an energy function. The energy function maybe represented using Equation 5:

$\begin{matrix}{{f_{L}(L)} = {{\sum\limits_{\partial_{*}}{\omega_{*}{{{K*{\partial_{*}L}} - {\partial_{*}B}}}^{2}}} + {\alpha {{\nabla\; L}}^{2}}}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

where ∂_(*)ε{∂_(α), ∂_(x), ∂_(y), ∂_(xx), ∂_(xy), ∂_(yy)} denotes apartial differentiation operator and an order in different directions,ω_(*)ε{ω₀, ω₁, ω₂} denotes a weight for each partial differentiation,and α denotes a weight (e.g., 0 or 1) for a regularization term.

The first term of Equation 5 is based on the blur model using imagederivatives proposed by Shan et al. (SHAN, Q., JIA, J., and AGARWALA,A., 2008, High-quality Motion Deblurring From A Single Image. ACM Trans.Graphics 27, 73.) in order to reduce ringing artifacts. Also, aregularization term ∥∇L∥² takes a latent image L having a smoothgradient.

Equation 5 may be optimized very quickly through pixel-wise division ina frequency domain using only two fast Fourier transforms. Anoptimization result of Equation 5 may include smooth edges and ringingartifacts. However, since only sharp edges are restored and otherportions including ringing artifacts are truncated in step 120, a resultof performing single deconvolution based on the prediction step 120 doesnot affect accurate blur kernel estimation.

In the method of deblurring an image according to an exemplaryembodiment of the present invention, a multi-scale iterative process isperformed to effectively deblur a large blurred image.

FIG. 4 illustrates kernels estimated in different scales in the methodof deblurring an image according to an exemplary embodiment of thepresent invention.

The multi-scale iterative process will be described with reference toFIG. 4. First, sharp edges are estimated in lower resolution in which arange of blur is narrow and the edges can be estimated without severelocalization errors.

In higher resolution, sharp edge estimation begins while avoiding thesevere localization errors from a deconvoluted image acquired throughthe previous lower resolution.

The estimation of the sharp edges during the iterative process in aspecific scale is applied to a deconvolution result obtained through theblur kernel updated in a previous iterative process.

In the method of deblurring an image according to an exemplaryembodiment of the present invention, a blur kernel for large blur may beestimated using small bidirectional and shock filters through themulti-scale iterative process, as described above.

The multi-scale iterative process enables estimation of large blur thathas not been handled using one scale. Also, in the method of deblurringan image according to an exemplary embodiment of the present invention,two fast Fourier transforms and simple image filtering are performed inpredicting the gradient map of the latent image for blur kernelestimation, and bidirectional filtering, shock filtering and gradientthreshold value application may be performed very quickly. The method isperformed faster than conventional deblurring methods (e.g., the methodproposed by Shan et al., 2008) that perform 30 to 60 fast Fouriertransforms.

The blur kernel estimation using the estimated gradient map {Px, Py}uses the energy function in Equation 6:

$\begin{matrix}{{{f_{K}(K)} = {{\sum\limits_{({P_{*},B_{*}})}{\omega_{*}{{{K*P_{*}} - B_{*}}}^{2}}} + {\beta {K}^{2}}}},} & {{Equation}\mspace{14mu} 6}\end{matrix}$

where ω_(*)ε{ω₁, ω₂} denotes a weight for each partial differentiation,and P_(*) and B_(*) may vary with Equation 7:

(P _(*) ,B _(*))ε{(P _(x),∂_(x) B),(P _(y),∂_(y) B), (∂_(x) P_(x),∂_(xx) B),(∂_(y) P _(y),∂_(yy) B), ((∂_(x) P _(y)+∂_(y) P_(x))/2,∂_(xy) B)}.  Equation 7

In Equation 6, respective (K*P_(*)−B_(*)) form a map I, which is definedas ∥I∥²=Σ_((x,y))I(x,y)², where (x, y) denotes an index of a pixel in I.β denotes a weight for Tikhonov regularization.

As shown in Equation 6, the present invention uses only imagederivatives without including a pixel value in the energy function. Thepresent invention uses a Tikhonov regularization term rather than L1norm of K in the conventional deblurring method.

Equation 6 may be represented in a matrix form as shown in Equation 8:

$\begin{matrix}\begin{matrix}{{f_{k}(k)} = {{{{Ak} - b}}^{2} + {\beta {k}^{2}}}} \\{= {{( {{Ak} - b} )^{T}( {{Ak} - b} )} + {\beta \; k^{T}k}}}\end{matrix} & {{Equation}\mspace{14mu} 8}\end{matrix}$

where A denotes a matrix consisting of five P_(*), and k denotes avector of the blur kernel K. b denotes a matrix consisting of fiveB_(*).

Using a conjugate gradient (CG) to simplify Equation 8, a gradient off_(k) is defined as shown in Equation 9:

$\begin{matrix}{{\frac{\partial{f_{k}(k)}}{\partial k} = {{2A^{T}{Ak}} + {2\; \beta \; k} - {2A^{T}b}}},} & {{Equation}\mspace{14mu} 9}\end{matrix}$

where calculation of ∂f_(k)(k)/∂k requires much time due to a size of A.When the latent image L and the blur kernel K have respective sizes ofn×n and m×m, the size of A becomes 5n²×m². Accordingly, directcalculation of Ak requires a very great amount of calculation and alarge storage capacity. Even though A^(T)A has a size of relativelysmall m²×m², it still requires much calculation time.

However, since Ak corresponds to convolution between five P_(*) and K,the calculation speed can be improved by using a fast Fourier transform.In particular, the Ak calculation requires six fast Fourier transforms,i.e., one

(K) and five

⁻¹[ω_(*)

(P_(*))∘

(K)]. Here,

and

⁻¹ denote a forward FFT and an inverse FFT, respectively, and ∘ denotespixel-wise multiplication.

(P_(*)) may be calculated earlier than the conjugate gradient (CG).

A calculation speed of A^(T)y (where y=Ak) can be improved through sixfast Fourier transforms, similar to the above-described method.

As a result, in each iteration step of the conjugate gradient (CG)method a total of 12 fast Fourier transforms are performed forcalculation of the gradient ∂f_(k)(k)/∂k. A pre-processing step maycalculate A^(T)b as well as

(P_(*)) through a fast Fourier transform.

In the present invention, A^(T)Ak directly calculated by associating Akwith A^(T)y in order to further increase the calculation speed byreducing the number of fast Fourier transforms. A^(T)Ak may berepresented using Equation 10:

$\begin{matrix}{\mathcal{F}^{- 1}\lbrack {\sum\limits_{P_{*}}{\omega_{*}{\overset{\_}{\mathcal{F}( P_{*} )} \cdot {\mathcal{F}( P_{*} )} \cdot {\mathcal{F}(K)}}}} \rbrack} & {{Equation}\mspace{14mu} 10}\end{matrix}$

where

denotes a complex conjugate of

(P_(*)).

$\sum\limits_{P_{*}}{\omega_{*}{\overset{\_}{\mathcal{F}( P_{*} )} \cdot {\mathcal{F}( P_{*} )}}}$

may be calculated in advance before the conjugate gradient (CG) methodis iteratively performed. Accordingly, fast Fourier transforms are onlyperformed twice for gradient calculation, such that a number of fastFourier transforms can be reduced by 10 times.

Efficient calculation as described above results from the use of onlythe image derivatives without using the pixel value in Equation 6. Thatis, since the present invention uses only the image partialdifferentiation, the boundary of the image is extended to a region wherea partial differentiation value is 0 prior to Ak calculation to avoid animage boundary issue.

After Equation 6 is optimized, values of elements smaller than 1/20 of amaximum value are set to 0 and other non-zero values are regularized sothat a sum of the non-zero values can become 1.

The iteration number and the convergence speed for optimizationconvergence are very critical in the numerical optimization process. Themethod of estimating a blur kernel according to an exemplary embodimentof the present invention achieves a faster calculation optimization thanthe conventional method using pixel values.

FIG. 5 illustrates a comparison in a convergence speed of blur kernelestimation between a method of estimating a blur kernel according to anexemplary embodiment of the present invention and a conventional methodof estimating a blur kernel.

It can be seen from FIG. 5 that the method of estimating a blur kernelaccording to an exemplary embodiment of the present invention greatlyreduces a blur kernel estimation error through iterations in a few timeswhile the conventional method of estimating a blur kernel using pixelvalues relatively slowly reduces the blur kernel estimation error.

As described above, fast convergence in numerical optimization resultsfrom the well-defined condition of the matrix A^(T)A in Equation 9.A^(T)A may be represented using Equation 11:

$\begin{matrix}{{A^{T}A} = {\sum\limits_{*}{\omega_{*}A_{*}^{T}A_{*}}}} & {{Equation}\mspace{14mu} 11}\end{matrix}$

where A_(*) ^(T) A_(*) is defined as (A_(*) ^(T)A_(*))_((i,j))=(I_(*)^(i))^(T)(*^(j)). Here, I_(*) ^(i) is a vector representation of ∂_(*)Lshifted by an amount dependent on i. Image partial differentiation isgenerally close to 0 except for edge pixels. Accordingly, A_(*)^(T)A_(*) has a great element value in a diagonal region and a verysmall element value in an off-diagonal region.

For comparison of blur kernel estimation speeds, the methods ofestimating a kernel are considered in a state where the energy functionis the same as that proposed by “Shan et al., 2008” except that aTikhonov regularization term is used in regularization terms for kernelestimation. Here, “Shan et al., 2008” using the Tikhonov regularizationterm is referred to as “Shan-L2”.

An original version of “Shan et al., 2008” requires a greater amount ofcalculation because it uses the L1 norm in the regularization terms ofthe energy function. Also, in the original version of “Shan et al.,2008”, a very large matrix is created in the kernel estimation step,which causes memory insufficiency and an inordinate calculation amount.

On the other hand, the blur kernel estimation according to an exemplaryembodiment of the present invention performs a fast Fourier transformtwice in performing each conjugate gradient (CG) method while “Shan-L2”performs a total of 14 fast Fourier transforms because it uses siximages: one pixel value image and five partially differentiated images.Also, “Shan-L2” requires a greater number of iterations than the presentinvention because the former uses the pixel value.

It can be seen from FIG. 5 that in the method of estimating a blurkernel according to an exemplary embodiment of the present inventionfive conjugate gradient (CG) processes were iteratively performed while“Shan-L2” requires 30 iterations to obtain similar accuracy.

As a result, the method of estimating a blur kernel according to anexemplary embodiment of the present invention is 40 times faster than“Shan-L2” and 10 or more times faster than other conventional methodsusing pixel values.

FIG. 6 illustrates the accuracy of blur kernel estimation according toan exemplary embodiment of the present invention.

The present invention and a conventional method using pixel values werecompared using a test image including different Gaussian noises toverify the accuracy of the method of estimating a kernel according to anexemplary embodiment of the present invention.

Referring to FIG. 6, the method of estimating a kernel according to anexemplary embodiment of the present invention has substantially the sameaccuracy as a conventional method of estimating a kernel using pixelvalues although the former does not use the pixel value. In someexemplary embodiments, the method of estimating a kernel according tothe present invention has a higher accuracy than the conventional methodof estimating a kernel using pixel values due to a well-definedcalculation system obtained by excluding the pixel value.

FIG. 7 illustrates images deblurred using a method of estimating blur ofan image according to an exemplary embodiment of the present invention.

It can be seen from FIG. 7 that an estimated blur kernel issubstantially the same as an original blur kernel and a deblurred imageincludes exactly restored details of the original image.

FIG. 8 illustrates actual images deblurred using a method of estimatingblur of an image according to an exemplary embodiment of the presentinvention. Respective images shown in FIG. 8 include different complexstructures and camera motions.

It can be seen from FIG. 8 that sharp edges of each deblurred image areconsiderably strengthened for a more prominent form and structure of anobject.

FIG. 9 illustrates a processing time for deblurring images shown in FIG.8.

For an experimental environment, a personal computer (PC) having an MSWindows XP (32 bits)-based Intel Core 2 CPU 2.66 GHz, 3.25 GB RAM, andan NVIDIA GeForce GTX graphic card was prepared. For the experiment, thedeblurring method was embodied using a GPU acceleration function of thegraphic card. It can be seen from FIG. 9 that even when a kernel islarge, an input blurred image can be deblurred within a few seconds bythe method of deblurring an image according to an exemplary embodimentof the present invention.

With the method of deblurring an image and the recording medium havingthe deblurring method recorded thereon, image filtering is applied to areceived blurred image to restore edges and eliminate noises, a gradientmap is predicted, the blur kernel is estimated based on the gradient mapand the blurred image, and deconvolution is performed on the estimatedblur kernel and the blurred image to estimate the latent image. Theprocesses are iteratively performed to improve estimation accuracy forthe blur kernel and then estimate a final blur kernel, and finaldeconvolution is performed on the final blur kernel and the blurredimage to deblur the image. The blur kernel estimation process uses onlyimage derivatives without using pixel values.

Thus, blur can be eliminated while maintaining the quality of the imagewith only one blurred image, and a deblurring time can be greatlyreduced, compared with conventional deblurring methods.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the present inventionwithout departing from the spirit or scope of the invention. Thus, it isintended that the present invention cover the modifications andvariations of this invention provided they come within the scope of theappended claims and their equivalents.

1. A method of deblurring an image, comprising: receiving a blurredimage; an image estimation step of estimating a non-blurred image fromthe blurred image; a blur information estimation step of estimating blurinformation from the blurred image and the estimated non-blurred image;and a deblurring step of deblurring the blurred image based on theblurred image and the estimated blur information, wherein the imageestimation step and the blur information estimation step are iterativelyperformed.
 2. The method of claim 1, wherein the image estimation stepcomprises: a gradient information prediction step of predicting gradientinformation for the blurred image; and a first deconvolution step ofperforming first deconvolution based on the estimated blur informationand the blurred image.
 3. The method of claim 2, wherein the gradientinformation prediction step comprises: applying bidirectional filteringto the blurred image; applying shock filtering to the bidirectionalfiltered image; calculating a gradient map for the shock filtered image;and applying a threshold value to the calculated gradient map.
 4. Themethod of claim 3, wherein applying the threshold value to thecalculated gradient map comprises: creating a histogram based on adirection and a size of the calculated gradient; setting a size of agradient capable of including as many pixels as a predetermined multipleor more of a maximum value of vertical and horizontal sizes of the blurinformation corresponding to each direction included in the createdhistogram, as a threshold value; and applying the set threshold value tothe gradient map to truncate the gradient.
 5. The method of claim 4,wherein applying the set threshold value to the gradient map to truncatethe gradient comprises setting a gradient smaller than the thresholdvalue to about zero.
 6. The method of claim 4, wherein the blurinformation estimation step comprises estimating the blur informationbased on the blurred image and the truncated gradient.
 7. The method ofclaim 4, wherein the blur information estimation step comprisesestimating the blur information using an energy function including onlyimage derivatives without a pixel value.
 8. The method of claim 1,wherein the image estimation step and the blur information estimationstep are iteratively performed while changing resolutions of the blurredimage and the estimated non-blurred image.
 9. A recording medium havinga program of instructions recorded thereon, wherein the program istangibly embodied and is readable and executable by a digital processingapparatus for deblurring a blurred image, the program performing:receiving a blurred image; an image estimation step of estimating anon-blurred image from the blurred image; a blur information estimationstep of estimating blur information from the blurred image and theestimated non-blurred image; and a deblurring step of deblurring theblurred image based on the blurred image and the estimated blurinformation, wherein the image estimation step and the blur informationestimation step are iteratively performed.